Disk spill with scaling improvements#486
Closed
gabrielbosio wants to merge 8 commits into
Closed
Conversation
Tables with effective width < 42 were sized at 2^21, producing single large chunks. Capping at 2^20 produces 2x more chunks of the standard size, improving parallel throughput and keeping peak memory per chunk uniform across all tables.
Tables sharing the same lde_size now reuse a single Arc<LdeTwiddles> instead of each allocating their own copy. This avoids redundant twiddle computation and allocation when multiple tables happen to have the same trace length (e.g., all 2^19-row tables after the cap change).
Under the `parallel` feature, fold each conjugate pair concurrently using `par_chunks(2).zip(inv_twiddles.par_iter())`, collecting into a new Vec of half length. The sequential path (no `parallel` feature) is unchanged. This avoids the index aliasing issue from an in-place interleaved write.
Replace the per-row `Vec` allocation inside the iterator with a thread-local buffer via Rayon's `map_init` (parallel path) or a single buffer allocated outside the loop (sequential path). This avoids one heap allocation per row during Merkle tree construction.
Collaborator
Author
|
/bench |
Benchmark — fib_iterative_8M (median of 3)Table parallelism: 32 (auto = cores / 3)
Commit: bddf96a · Baseline: built from main · Runner: self-hosted bench |
gabrielbosio
marked this pull request as ready for review
April 9, 2026 20:55
… into perf/disk-spill-time-regression-scaling
Collaborator
Author
|
/bench |
jotabulacios
approved these changes
Apr 14, 2026
Collaborator
Author
|
Since #497 exists, I think it doesn't make sense to keep this PR open. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A merge of #476 with the first 5 commits of #483 to check how heap peak gets reduced.